Digital pixel array with adaptive exposure

ABSTRACT

Methods and systems for light sensing are provided. In one example, an apparatus comprises and an array of pixel cells and a controller. Each pixel cell of the array of pixel cells includes a photodiode configured to generate charges upon receiving incident light and a capacitor configured to accumulate the charges generated by the photodiode. The controller is configured to: start an exposure period to accumulate the charges at the pixel cells; and based on a determination that the quantity of charges accumulated by the at least one pixel cell exceeds a pre-determined threshold: end the exposure period to cause the capacitors of the array of pixel cells to stop accumulating the charges, generate an output pixel value for each pixel cell based on the charges accumulated at the capacitor of the each pixel cell within the exposure period; and provide the output pixel values to generate an image frame.

RELATED APPLICATION

This patent application claims priority to U.S. Provisional PatentApplication Ser. No. 62/637,970, filed Mar. 2, 2018, entitled “DigitalPixel Array With Adaptive Exposure,” which is assigned to the assigneehereof and is incorporated herein by reference in its entirety for allpurposes.

BACKGROUND

The disclosure relates generally to image sensors, and more specificallyto pixel cell structure including interfacing circuitries fordetermining light intensity for image generation.

A typical image sensor includes a photodiode to sense incident light byconverting photons into charges (e.g., electrons or holes). The imagesensor further includes a floating node configured as a capacitor tocollect the charges generated by the photodiode during an exposureperiod. The collected charges can develop a voltage at the capacitor.The voltage can be buffered and fed to an analog-to-digital converter(ADC), which can convert the voltage into a digital value representingthe intensity of the incident light.

SUMMARY

The present disclosure relates to image sensors. More specifically, andwithout limitation, this disclosure relates to a pixel cell array thatsupports multi-stage readouts in an exposure period to generate an imageframe.

In one example, an apparatus is provided. The apparatus comprises anarray of pixel cells, each pixel cell including a photodiode configuredto generate charges upon receiving incident light and a capacitorconfigured to accumulate the charges generated by the photodiode. Theapparatus further comprises a controller configured to: start anexposure period to enable the capacitors of the array of pixel cells toaccumulate the charges; determine whether a quantity of chargesaccumulated by at least one pixel cell of the array of pixel cellsexceeds a pre-determined threshold; and based on a determination thatthe quantity of charges accumulated by the at least one pixel cellexceeds a pre-determined threshold: end the exposure period to cause thecapacitors of the array of pixel cells to stop accumulating the charges,generate an output pixel value for each pixel cell based on the chargesaccumulated at the capacitor of the each pixel cell within the exposureperiod; and provide the output pixel values for generation of an imageframe.

In some aspects, the controller is further configured to: determine anintermediate pixel value for each pixel cell based on the chargesaccumulated at the capacitor of the each pixel cell within the exposureperiod; determine a scale value based on a duration of the endedexposure period; and scale each of the intermediate pixel values usingthe scale value to generate the output pixel values.

In some aspects, the pre-determined threshold is set based on anintensity range of incident light that saturates the at least one pixelcell. In some aspects, the pre-determined threshold is set based on acapacity of the capacitor of the at least one pixel cell foraccumulating the charges.

In some aspects, the apparatus further comprises one or moreanalog-to-digital converters (ADC) configured to generate a digitalpixel value based on at least one of: a measurement of time for thecapacitor of a pixel cell to accumulate a quantity of charges equal tothe pre-determined threshold, or a measurement of the quantity ofcharges accumulated at the capacitor when the exposure period ends; anda selection module configured to couple each pixel cell of the array ofpixel cells sequentially to the one or more ADCs to generate the digitalpixel value for the each pixel cell based on the charges accumulated atthe photodiode of the each pixel cell.

In some aspects, the controller is configured to, in each exposureperiod of a plurality of exposure periods: select a pixel cell from thearray of the pixel cells as the at least one pixel cell; control theselection module to couple the selected pixel cell to the one or moreADCs to determine whether a quantity of charges accumulated at theselected pixel cell exceeds the pre-determined threshold; and responsiveto determining that the quantity of charges accumulated at the selectedpixel cell exceeds the pre-determined threshold, end the each exposureperiod.

In some aspects, the controller is configured to select the same pixelcell in the each exposure period of the plurality of exposure periods.In some aspects, the controller is configured to select different pixelcells in a first exposure period and a second exposure period of theplurality of exposure periods. In some aspects, the controller is alsoconfigured to select the pixel cell in a current exposure period basedon the digital pixel value of the pixel cell exceeding thepre-determined threshold in a prior exposure period. In some aspects,the controller is configured to select the pixel cell based on a randomfunction.

In some aspects, each pixel cell of the apparatus may include ananalog-to-digital converter (ADC) configured to generate a digital pixelvalue for the each pixel cell based on at least one of: a measurement oftime for the capacitor of the each pixel cell to accumulate a quantityof charges equal to the pre-determined threshold, or a measurement ofthe quantity of charges accumulated at the capacitor when the exposureperiod ends. The controller may monitor for an indication that aquantity of charges accumulated at at least one of the pixel cellsexceeds the pre-determined threshold; and end the exposure period foreach pixel cell based on receiving the indication.

In some aspects, the exposure period may have a default end time. Thecontroller may end the exposure period before the default end time basedon the determination that the quantity of charges accumulated by the atleast one pixel cell exceeds the pre-determined threshold. The defaultend time may be preset based on an ambient light intensity.

In some aspects, the array of pixel cells of the apparatus is a firstarray of pixel cells. The apparatus may further comprise a second arrayof pixel cells. The controller may start the exposure period at a firsttime for the first array and for the second array; end the exposureperiod at a second time for the first array; and end the exposure periodat a third time different from the second time for the second array.

In one example, a method is provided. The method may comprise: startingan exposure period to enable a capacitor of each pixel cell of an arrayof pixel cells to accumulate charges generated by a photodiode includedin the each pixel cell; determining whether a quantity of chargesaccumulated by at least one pixel cell of the array of pixel cellsexceeds a pre-determined threshold; and based on determining that thequantity of charges accumulated by the at least one pixel cell exceeds apre-determined threshold: ending the exposure period to cause thecapacitors of the array of pixel cells to stop accumulating the charges,generating an output pixel value for each pixel cell based on thecharges accumulated at the capacitor of the each pixel cell within theexposure period; and providing the output pixel values for generation ofan image frame.

In some aspects, the method further comprise: determining anintermediate pixel value for each pixel cell based on the chargesaccumulated at the capacitor of the each pixel cell within the exposureperiod; determining a scale value based on a duration of the endedexposure period; and scaling each of the intermediate pixel values usingthe scale value to generate the output pixel values.

In some aspects, the method further comprises: generating, using an ADC,a digital pixel value for each pixel cell based on at least one of: ameasurement of time for the capacitor of the each pixel cell toaccumulate a quantity of charges equal to the pre-determined threshold,or a measurement of the quantity of charges accumulated at the capacitorof the each pixel cell when the exposure period ends.

In some aspects, the method further comprises, in each exposure periodof a plurality of exposure periods: selecting a pixel cell from thearray of the pixel cells as the at least one pixel cell; controlling theADC to determine whether a quantity of charges accumulated at theselected pixel cell exceeds the pre-determined threshold; and responsiveto determining that the quantity of charges accumulated at the selectedpixel cell exceeds the pre-determined threshold, ending the eachexposure period. In some aspects, the same pixel cell is selected in theeach exposure period of the plurality of exposure periods. In someaspects, different pixel cells in a first exposure period are selectedin a first exposure period and in a second exposure period of theplurality of exposure periods. In some aspects, the pixel cell isselected in a current exposure period based on the digital pixel valueof the pixel cell exceeding the pre-determined threshold in a priorexposure period. In some aspects, the pixel cell is selected based on arandom function.

BRIEF DESCRIPTION OF THE DRAWINGS

Illustrative embodiments are described with reference to the followingfigures.

FIGS. 1A and 1B are diagrams of an embodiment of a near-eye display.

FIG. 2 is an embodiment of a cross section of the near-eye display.

FIG. 3 illustrates an isometric view of an embodiment of a waveguidedisplay with a single source assembly.

FIG. 4 illustrates a cross section of an embodiment of the waveguidedisplay.

FIG. 5 is a block diagram of an embodiment of a system including thenear-eye display.

FIGS. 6A and 6B illustrate examples of exposure time adjustment whichcan be performed by near-eye display of FIG. 5.

FIGS. 7A and 7B illustrate block diagrams of embodiments of an imagesensor.

FIG. 8 illustrates operations for determining light intensities ofdifferent ranges by embodiments of FIGS. 7A and 7B.

FIG. 9 illustrates examples of internal components of the pixel cell ofFIG. 7.

FIGS. 10A, 10B, 10C, and 10D illustrate techniques for quantizing alight intensity.

FIG. 11 illustrates block diagrams of an embodiment of a pixel cell.

FIGS. 12A, 12B, 12C, and 12D illustrate example methods for determininglight intensity.

FIG. 13 illustrates another example methods for determining lightintensity.

FIG. 14 illustrates an embodiment of a flowchart of a process fordetermining a light intensity.

FIG. 15 illustrates an embodiment of a flowchart of a process foroperating a pixel cell array.

The figures depict embodiments of the present disclosure for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated may be employed without departing from theprinciples, or benefits touted, of this disclosure.

In the appended figures, similar components and/or features may have thesame reference label. Further, various components of the same type maybe distinguished by following the reference label by a dash and a secondlabel that distinguishes among the similar components. If only the firstreference label is used in the specification, the description isapplicable to any one of the similar components having the same firstreference label irrespective of the second reference label.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, specificdetails are set forth in order to provide a thorough understanding ofcertain inventive embodiments. However, it will be apparent that variousembodiments may be practiced without these specific details. The figuresand description are not intended to be restrictive.

A typical image sensor includes a photodiode to sense incident light byconverting photons into charges (e.g., electrons or holes). The imagesensor further includes a floating node configured as a capacitor tocollect the charges generated by the photodiode during an exposureperiod. The collected charges can develop a voltage at the capacitor.The voltage can be buffered and fed to an analog-to-digital converter(ADC), which can convert the voltage into digital pixel datarepresenting the intensity of the incident light. Each of the digitalvalues has a bit depth (e.g., 8 bits, 16 bits, etc.) determined based onthe quantization resolution of the ADC.

An image can be generated based on intensity data provided by an arrayof image sensors, with each image sensor forming a pixel cell thatcorresponds to a pixel of the image. The array of pixel cells can bearranged into rows and columns, with each pixel cell generating avoltage representing the intensity for a pixel associated with aparticular location in the image. A number of pixels included in thearray can determine a resolution of the generated image. An image can bereconstructed based on the digital intensity data of each pixelgenerated by the ADC. The digital intensity data of each pixel of animage frame can be stored in a frame buffer for subsequent processing.

The digital value generated by the ADC, which reflects a number ofcharges stored at the floating node within an exposure period, maycorrelate to the intensity of the incident light. However, the degree ofcorrelation can be affected by different factors. For example, thequantity of charges stored in the floating node can be directly relatedto the intensity of the incident light until the floating node reaches asaturation limit. Beyond the saturation limit, the floating node may beunable to accumulate additional charges generated by the photodiode, andthe additional charges may be leaked and not stored. As a result, thequantity of the charges stored at the floating node may be lower thanthe quantity of charges actually generated by the photodiode. Thesaturation limit may determine an upper limit of the measurable lightintensity of the image sensor, and an image sensor may become inoperable(or generate a low quality image) in an environment with strong ambientlight.

Moreover, saturating the floating nodes of the pixel cells may causeother effects that can further degrade the quality of imaging, such asblooming. Blooming occurs when the image sensor is exposed to a highintensity light source (e.g., a light bulb) in a scene, which cansaturate the floating nodes of some but not all of the pixel cells.Charges may leak from the saturated pixel cells into other neighboringpixel cells and contaminate the charges stored in those neighboringpixel cells. The contamination can degrade the correlation between theincident light intensity and the accumulated charges in thoseneighboring pixel cells. The leakage can create an effect of a blanketof light that obscures the rest of the scene in the image and degradesthe imaging of the scene. Because of the effect of saturation andblooming, the image sensor may also become inoperable to capture theimage of a scene including a high intensity light source.

Image sensors can be found in many different applications. As anexample, image sensors are included in digital imaging devices (e.g.,digital cameras, smart phones, etc.) to provide digital imaging. Asanother example, image sensors can be configured as input devices tocontrol or influence the operation of a device, such as controlling orinfluencing the display content of a near-eye display in wearablevirtual-reality (VR) systems and/or augmented-reality (AR) and/or mixedreality (MR) systems. For example, the image sensors can be used togenerate physical image data of a physical environment in which a useris located. The physical image data can be provided to a locationtracking system. A location tracking system can obtain the digitalintensity data of an image frame from the frame buffer and search forpatterns of intensity data representing certain image features toidentify one or more physical objects in the image. The image locationsof the physical objects can be tracked to determine the locations of thephysical objects in the environment (e.g., through triangulation). Thelocation information of the physical objects can be provided to asimultaneous localization and mapping (SLAM) algorithm to determine, forexample, a location of the user, an orientation of the user, and/or apath of movement of the user in the physical environment. The image datacan also be used to, for example, generate stereo depth information formeasuring a distance between the user and the physical object in thephysical environment, to track a movement of the physical object (e.g.,user's eyeballs), etc. In all these examples, the VR/AR/MR system cangenerate and/or update virtual image data based on the information(e.g., a location of a user, a gaze point direction, etc.) obtained fromthe physical image data, and provide the virtual image data fordisplaying to the user via the near-eye display to provide aninteractive experience.

A wearable VR/AR/MR system may operate in environments with a very widerange of light intensities. For example, the wearable VR/AR/MR systemmay be able to operate in an indoor environment or in an outdoorenvironment, and/or at different times of the day, and the lightintensity of the operation environment of the wearable VR/AR/MR systemmay vary substantially. Moreover, the wearable VR/AR/MR system may alsoinclude the aforementioned NIR eyeball tracking system, which mayrequire projecting lights of very low intensity into the eyeballs of theuser to prevent damaging the eyeballs. As a result, the image sensors ofthe wearable VR/AR/MR system may need to have a wide dynamic range to beable to operate properly (e.g., to generate an output that correlateswith the intensity of incident light) across a very wide range of lightintensities associated with different operating environments. The imagesensors of the wearable VR/AR/MR system may also need to generate imagesat sufficient high speed to allow tracking of the user's location,orientation, gaze point, etc. Image sensors with relatively limiteddynamic ranges and which generate images at relatively low speed may notbe suitable for such a wearable VR/AR/MR system. However, as describedabove, the effects of saturation and blooming on image sensing may limitthe applications of the wearable VR/AR/MR system in an environment ofstrong ambient light or in an environment with high intensity lightsources, which can degrade user experience.

This disclosure relates to an image sensor. The image sensor may includean array of pixel cells, with each pixel cell including a photodiodeconfigured to generate charges upon receiving incident light, and acapacitor configured to accumulate the charges generated by thephotodiode. The image sensor may further include a controller configuredto start an exposure period to enable the capacitors of the array ofpixel cells to accumulate the charges, and determine whether a quantityof charges accumulated by at least one pixel cell of the array of pixelcells exceed a pre-determined threshold. Based on a determination thatthe quantity of charges accumulated by the at least one pixel cellexceeds a pre-determined threshold, the controller can stop the exposureperiod to cause the capacitors of the array of pixel cells to stopaccumulating the charges, and generate an output pixel value for eachpixel cell based on the charges accumulated at the capacitor of the eachpixel cell within the exposure period. The output pixel values can beprovided to, for example, an image processor to generate an image frame.

The disclosed techniques can dynamically adjust an exposure period for apixel cell array to, for example, reduce the likelihood of saturatingthe pixel cells in a case where the pixel cell array is exposed to highintensity light, which also reduces the likelihood of blooming.Moreover, the dynamic adjustment can be performed within the exposureperiod of a single image frame, which enables the adjustment to be moreresponsive to a change in the intensity of light received by the pixelcell array caused by, for example, a movement of objects and/or lightsources in the environment, a movement of the wearable VR/AR/MR systemthat incorporates the pixel cell array, etc. All these can extend thedynamic range of the pixel cell array and enable the wearable VR/AR/MRsystem to perform accurate image sensing for a wider range ofapplications, thereby improving user experience.

Embodiments of the disclosure may include or be implemented inconjunction with an artificial reality system. Artificial reality is aform of reality that has been adjusted in some manner beforepresentation to a user, which may include, e.g., a virtual reality (VR),an augmented reality (AR), a mixed reality (MR), a hybrid reality, orsome combination and/or derivatives thereof. Artificial reality contentmay include completely generated content or generated content combinedwith captured (e.g., real-world) content. The artificial reality contentmay include video, audio, haptic feedback, or some combination thereof,any of which may be presented in a single channel or in multiplechannels (such as stereo video that produces a three-dimensional effectto the viewer). Additionally, in some embodiments, artificial realitymay also be associated with applications, products, accessories,services, or some combination thereof, that are used to, e.g., createcontent in an artificial reality and/or are otherwise used in (e.g.,perform activities in) an artificial reality. The artificial realitysystem that provides the artificial reality content may be implementedon various platforms, including a head-mounted display (HMD) connectedto a host computer system, a standalone HMD, a mobile device orcomputing system, or any other hardware platform capable of providingartificial reality content to one or more viewers.

FIG. 1A is a diagram of an embodiment of a near-eye display 100.Near-eye display 100 presents media to a user. Examples of mediapresented by near-eye display 100 include one or more images, video,and/or audio. In some embodiments, audio is presented via an externaldevice (e.g., speakers and/or headphones) that receives audioinformation from the near-eye display 100, a console, or both, andpresents audio data based on the audio information. Near-eye display 100is generally configured to operate as a virtual reality (VR) display. Insome embodiments, near-eye display 100 is modified to operate as anaugmented reality (AR) display and/or a mixed reality (MR) display.

Near-eye display 100 includes a frame 105 and a display 110. Frame 105is coupled to one or more optical elements. Display 110 is configuredfor the user to see content presented by near-eye display 100. In someembodiments, display 110 comprises a waveguide display assembly fordirecting light from one or more images to an eye of the user.

Near-eye display 100 further includes image sensors 120 a, 120 b, 120 c,and 120 d. Each of image sensors 120 a, 120 b, 120 c, and 120 d mayinclude a pixel array configured to generate image data representingdifferent fields of views along different directions. For example,sensors 120 a and 120 b may be configured to provide image datarepresenting two fields of view towards a direction A along the Z axis,whereas sensor 120 c may be configured to provide image datarepresenting a field of view towards a direction B along the X axis, andsensor 120 d may be configured to provide image data representing afield of view towards a direction C along the X axis.

In some embodiments, sensors 120 a-120 d can be configured as inputdevices to control or influence the display content of the near-eyedisplay 100, to provide an interactive VR/AR/MR experience to a user whowears near-eye display 100. For example, sensors 120 a-120 d cangenerate physical image data of a physical environment in which the useris located. The physical image data can be provided to a locationtracking system to track a location and/or a path of movement of theuser in the physical environment. A system can then update the imagedata provided to display 110 based on, for example, the location andorientation of the user, to provide the interactive experience. In someembodiments, the location tracking system may operate a SLAM algorithmto track a set of objects in the physical environment and within a viewof field of the user as the user moves within the physical environment.The location tracking system can construct and update a map of thephysical environment based on the set of objects, and track the locationof the user within the map. By providing image data corresponding tomultiple fields of views, sensors 120 a-120 d can provide the locationtracking system a more holistic view of the physical environment, whichcan lead to more objects to be included in the construction and updatingof the map. With such arrangement, the accuracy and robustness oftracking a location of the user within the physical environment can beimproved.

In some embodiments, near-eye display 100 may further include one ormore active illuminators 130 to project light into the physicalenvironment. The light projected can be associated with differentfrequency spectrums (e.g., visible light, infra-red light, ultra-violetlight, etc.), and can serve various purposes. For example, illuminator130 may project light in a dark environment (or in an environment withlow intensity of infra-red light, ultra-violet light, etc.) to assistsensors 120 a-120 d in capturing images of different objects within thedark environment to, for example, enable location tracking of the user.Illuminator 130 may project certain markers onto the objects within theenvironment, to assist the location tracking system in identifying theobjects for map construction/updating.

In some embodiments, illuminator 130 may also enable stereoscopicimaging. For example, one or more of sensors 120 a or 120 b can includeboth a first pixel array for visible light sensing and a second pixelarray for infra-red (IR) light sensing. The first pixel array can beoverlaid with a color filter (e.g., a Bayer filter), with each pixel ofthe first pixel array being configured to measure intensity of lightassociated with a particular color (e.g., one of red, green or bluecolors). The second pixel array (for IR light sensing) can also beoverlaid with a filter that allows only IR light through, with eachpixel of the second pixel array being configured to measure intensity ofIR lights. The pixel arrays can generate an RGB image and an IR image ofan object, with each pixel of the IR image being mapped to each pixel ofthe RGB image. Illuminator 130 may project a set of IR markers on theobject, the images of which can be captured by the IR pixel array. Basedon a distribution of the IR markers of the object as shown in the image,the system can estimate a distance of different parts of the object fromthe IR pixel array, and generate a stereoscopic image of the objectbased on the distances. Based on the stereoscopic image of the object,the system can determine, for example, a relative position of the objectwith respect to the user, and can update the image data provided todisplay 100 based on the relative position information to provide theinteractive experience.

As discussed above, near-eye display 100 may be operated in environmentsassociated with a very wide range of light intensities. For example,near-eye display 100 may be operated in an indoor environment or in anoutdoor environment, and/or at different times of the day. Near-eyedisplay 100 may also operate with or without active illuminator 130being turned on. As a result, image sensors 120 a-120 d may need to havea wide dynamic range to be able to operate properly (e.g., to generatean output that correlates with the intensity of incident light) across avery wide range of light intensities associated with different operatingenvironments for near-eye display 100.

FIG. 1B is a diagram of another embodiment of near-eye display 100. FIG.1B illustrates a side of near-eye display 100 that faces the eyeball(s)135 of the user who wears near-eye display 100. As shown in FIG. 1B,near-eye display 100 may further include a plurality of illuminators 140a, 140 b, 140 c, 140 d, 140 e, and 140 f. Near-eye display 100 furtherincludes a plurality of image sensors 150 a and 150 b. Illuminators 140a, 140 b, and 140 c may emit lights of certain frequency range (e.g.,NIR) towards direction D (which is opposite to direction A of FIG. 1A).The emitted light may be associated with a certain pattern, and can bereflected by the left eyeball of the user. Sensor 150 a may include apixel array to receive the reflected light and generate an image of thereflected pattern. Similarly, illuminators 140 d, 140 e, and 140 f mayemit NIR lights carrying the pattern. The NIR lights can be reflected bythe right eyeball of the user, and may be received by sensor 150 b.Sensor 150 b may also include a pixel array to generate an image of thereflected pattern. Based on the images of the reflected pattern fromsensors 150 a and 150 b, the system can determine a gaze point of theuser, and update the image data provided to display 100 based on thedetermined gaze point to provide an interactive experience to the user.

As discussed above, to avoid damaging the eyeballs of the user,illuminators 140 a, 140 b, 140 c, 140 d, 140 e, and 140 f are typicallyconfigured to output lights of very low intensities. In a case whereimage sensors 150 a and 150 b comprise the same sensor devices as imagesensors 120 a-120 d of FIG. 1A, the image sensors 120 a-120 d may needto be able to generate an output that correlates with the intensity ofincident light when the intensity of the incident light is very low,which may further increase the dynamic range requirement of the imagesensors.

Moreover, the image sensors 120 a-120 d may need to be able to generatean output at a high speed to track the movements of the eyeballs. Forexample, a user's eyeball can perform a very rapid movement (e.g., asaccade movement) in which there can be a quick jump from one eyeballposition to another. To track the rapid movement of the user's eyeball,image sensors 120 a-120 d need to generate images of the eyeball at highspeed. For example, the rate at which the image sensors generate animage frame (the frame rate) needs to at least match the speed ofmovement of the eyeball. The high frame rate requires short totalexposure period for all of the pixel cells involved in generating theimage frame, as well as high speed for converting the sensor outputsinto digital values for image generation. Moreover, as discussed above,the image sensors also need to be able to operate at an environment withlow light intensity.

FIG. 2 is an embodiment of a cross section 200 of near-eye display 100illustrated in FIG. 1A and FIG. 1B. Display 110 includes at least onewaveguide display assembly 210. An exit pupil 230 is a location where asingle eye 220 of the user is positioned in an eyebox region when theuser wears the near-eye display 100. For purposes of illustration, FIG.2 shows the cross section 200 with eye 220 and a single waveguidedisplay assembly 210, but a second waveguide display is used for asecond eye of a user.

Waveguide display assembly 210 is configured to direct image light to aneyebox located at exit pupil 230 and to eye 220. Waveguide displayassembly 210 may be composed of one or more materials (e.g., plastic,glass, etc.) with one or more refractive indices. In some embodiments,near-eye display 100 includes one or more optical elements betweenwaveguide display assembly 210 and eye 220.

In some embodiments, waveguide display assembly 210 includes a stack ofone or more waveguide displays including, but not restricted to, astacked waveguide display, a varifocal waveguide display, etc. Thestacked waveguide display is a polychromatic display (e.g., ared-green-blue (RGB) display) created by stacking waveguide displayswhose respective monochromatic sources are of different colors. Thestacked waveguide display is also a polychromatic display that can beprojected on multiple planes (e.g., multi-planar colored display). Insome configurations, the stacked waveguide display is a monochromaticdisplay that can be projected on multiple planes (e.g., multi-planarmonochromatic display). The varifocal waveguide display is a displaythat can adjust a focal position of image light emitted from thewaveguide display. In alternate embodiments, waveguide display assembly210 may include the stacked waveguide display and the varifocalwaveguide display.

FIG. 3 illustrates an isometric view of an embodiment of a waveguidedisplay 300. In some embodiments, waveguide display 300 is a component(e.g., waveguide display assembly 210) of near-eye display 100. In someembodiments, waveguide display 300 is part of some other near-eyedisplay or other system that directs image light to a particularlocation.

Waveguide display 300 includes a source assembly 310, an outputwaveguide 320, and a controller 330. For purposes of illustration, FIG.3 shows the waveguide display 300 associated with a single eye 220, butin some embodiments, another waveguide display separate, or partiallyseparate, from the waveguide display 300 provides image light to anothereye of the user.

Source assembly 310 generates image light 355. Source assembly 310generates and outputs image light 355 to a coupling element 350 locatedon a first side 370-1 of output waveguide 320. Output waveguide 320 isan optical waveguide that outputs expanded image light 340 to an eye 220of a user. Output waveguide 320 receives image light 355 at one or morecoupling elements 350 located on the first side 370-1 and guidesreceived input image light 355 to a directing element 360. In someembodiments, coupling element 350 couples the image light 355 fromsource assembly 310 into output waveguide 320. Coupling element 350 maybe, e.g., a diffraction grating, a holographic grating, one or morecascaded reflectors, one or more prismatic surface elements, and/or anarray of holographic reflectors.

Directing element 360 redirects the received input image light 355 todecoupling element 365 such that the received input image light 355 isdecoupled out of output waveguide 320 via decoupling element 365.Directing element 360 is part of, or affixed to, first side 370-1 ofoutput waveguide 320. Decoupling element 365 is part of, or affixed to,second side 370-2 of output waveguide 320, such that directing element360 is opposed to the decoupling element 365. Directing element 360and/or decoupling element 365 may be, e.g., a diffraction grating, aholographic grating, one or more cascaded reflectors, one or moreprismatic surface elements, and/or an array of holographic reflectors.

Second side 370-2 represents a plane along an x-dimension and ay-dimension. Output waveguide 320 may be composed of one or morematerials that facilitate total internal reflection of image light 355.Output waveguide 320 may be composed of e.g., silicon, plastic, glass,and/or polymers. Output waveguide 320 has a relatively small formfactor. For example, output waveguide 320 may be approximately 50 mmwide along x-dimension, 30 mm long along y-dimension and 0.5-1 mm thickalong a z-dimension.

Controller 330 controls scanning operations of source assembly 310. Thecontroller 330 determines scanning instructions for the source assembly310. In some embodiments, the output waveguide 320 outputs expandedimage light 340 to the user's eye 220 with a large field of view (FOV).For example, the expanded image light 340 is provided to the user's eye220 with a diagonal FOV (in x and y) of 60 degrees and/or greater and/or150 degrees and/or less. The output waveguide 320 is configured toprovide an eyebox with a length of 20 mm or greater and/or equal to orless than 50 mm; and/or a width of 10 mm or greater and/or equal to orless than 50 mm.

Moreover, controller 330 also controls image light 355 generated bysource assembly 310, based on image data provided by image sensor 370.Image sensor 370 may be located on first side 370-1 and may include, forexample, image sensors 120 a-120 d of FIG. 1A to generate image data ofa physical environment in front of the user (e.g., for locationdetermination). Image sensor 370 may also be located on second side370-2 and may include image sensors 150 a and 150 b of FIG. 1B togenerate image data of eye 220 (e.g., for gaze point determination) ofthe user. Image sensor 370 may interface with control circuitries thatare not located within waveguide display 300 (e.g., in a remoteconsole). Image sensor 370 may provide image data to the controlcircuitries, which may determine, for example, a location of the user, agaze point of the user, etc., and determine the content of the images tobe displayed to the user. The control circuitries can transmitinstructions to controller 330 related to the determined content. Basedon the instructions, controller 330 can control the generation andoutputting of image light 355 by source assembly 310.

FIG. 4 illustrates an embodiment of a cross section 400 of the waveguidedisplay 300. The cross section 400 includes source assembly 310, outputwaveguide 320, and image sensor 370. In the example of FIG. 4, imagesensor 370 may include a set of pixel cells 402 located on first side370-1 to generate an image of the physical environment in front of theuser. In some embodiments, there can be a mechanical shutter 404interposed between the set of pixel cells 402 and the physicalenvironment to control the exposure of the set of pixel cells 402. Insome embodiments, the mechanical shutter 404 can be replaced by anelectronic shutter gate, as to be discussed below. Each of pixel cells402 may correspond to one pixel of the image. Although not shown in FIG.4, it is understood that each of pixel cells 402 may also be overlaidwith a filter to control the frequency range of the light to be sensedby the pixel cells.

After receiving instructions from the control circuitries, mechanicalshutter 404 can open and expose the set of pixel cells 402 in anexposure period. During the exposure period, image sensor 370 can obtainsamples of lights incident on the set of pixel cells 402, and generateimage data based on an intensity distribution of the incident lightsamples detected by the set of pixel cells 402. Image sensor 370 canthen provide the image data to the remote console, which determines thedisplay content, and provide the display content information tocontroller 330. Controller 330 can then determine image light 355 basedon the display content information.

Source assembly 310 generates image light 355 in accordance withinstructions from the controller 330. Source assembly 310 includes asource 410 and an optics system 415. Source 410 is a light source thatgenerates coherent or partially coherent light. Source 410 may be, e.g.,a laser diode, a vertical cavity surface emitting laser, and/or a lightemitting diode.

Optics system 415 includes one or more optical components that conditionthe light from source 410. Conditioning light from source 410 mayinclude, e.g., expanding, collimating, and/or adjusting orientation inaccordance with instructions from controller 330. The one or moreoptical components may include one or more lenses, liquid lenses,mirrors, apertures, and/or gratings. In some embodiments, optics system415 includes a liquid lens with a plurality of electrodes that allowsscanning of a beam of light with a threshold value of scanning angle toshift the beam of light to a region outside the liquid lens. Lightemitted from the optics system 415 (and also source assembly 310) isreferred to as image light 355.

Output waveguide 320 receives image light 355. Coupling element 350couples image light 355 from source assembly 310 into output waveguide320. In embodiments where coupling element 350 is diffraction grating, apitch of the diffraction grating is chosen such that total internalreflection occurs in output waveguide 320, and image light 355propagates internally in output waveguide 320 (e.g., by total internalreflection), toward decoupling element 365.

Directing element 360 redirects image light 355 toward decouplingelement 365 for decoupling from output waveguide 320. In embodimentswhere directing element 360 is a diffraction grating, the pitch of thediffraction grating is chosen to cause incident image light 355 to exitoutput waveguide 320 at angle(s) of inclination relative to a surface ofdecoupling element 365.

In some embodiments, directing element 360 and/or decoupling element 365are structurally similar. Expanded image light 340 exiting outputwaveguide 320 is expanded along one or more dimensions (e.g., may beelongated along x-dimension). In some embodiments, waveguide display 300includes a plurality of source assemblies 310 and a plurality of outputwaveguides 320. Each of source assemblies 310 emits a monochromaticimage light of a specific band of wavelength corresponding to a primarycolor (e.g., red, green, or blue). Each of output waveguides 320 may bestacked together with a distance of separation to output an expandedimage light 340 that is multi-colored.

FIG. 5 is a block diagram of an embodiment of a system 500 including thenear-eye display 100. The system 500 comprises near-eye display 100, animaging device 535, an input/output interface 540, and image sensors 120a-120 d and 150 a-150 b that are each coupled to control circuitries510. System 500 can be configured as a head-mounted device, a wearabledevice, etc.

Near-eye display 100 is a display that presents media to a user.Examples of media presented by the near-eye display 100 include one ormore images, video, and/or audio. In some embodiments, audio ispresented via an external device (e.g., speakers and/or headphones) thatreceives audio information from near-eye display 100 and/or controlcircuitries 510 and presents audio data based on the audio informationto a user. In some embodiments, near-eye display 100 may also act as anAR eyewear glass. In some embodiments, near-eye display 100 augmentsviews of a physical, real-world environment, with computer-generatedelements (e.g., images, video, sound, etc.).

Near-eye display 100 includes waveguide display assembly 210, one ormore position sensors 525, and/or an inertial measurement unit (IMU)530. Waveguide display assembly 210 includes source assembly 310, outputwaveguide 320, and controller 330.

IMU 530 is an electronic device that generates fast calibration dataindicating an estimated position of near-eye display 100 relative to aninitial position of near-eye display 100 based on measurement signalsreceived from one or more of position sensors 525.

Imaging device 535 may generate image data for various applications. Forexample, imaging device 535 may generate image data to provide slowcalibration data in accordance with calibration parameters received fromcontrol circuitries 510. Imaging device 535 may include, for example,image sensors 120 a-120 d of FIG. 1A for generating image data of aphysical environment in which the user is located, for performinglocation tracking of the user. Imaging device 535 may further include,for example, image sensors 150 a-150 b of FIG. 1B for generating imagedata for determining a gaze point of the user, for identifying an objectof interest of the user, etc.

The input/output interface 540 is a device that allows a user to sendaction requests to the control circuitries 510. An action request is arequest to perform a particular action. For example, an action requestmay be to start or end an application or to perform a particular actionwithin the application.

Control circuitries 510 provides media to near-eye display 100 forpresentation to the user in accordance with information received fromone or more of: imaging device 535, near-eye display 100, andinput/output interface 540. In some examples, control circuitries 510can be housed within system 500 configured as a head-mounted device. Insome examples, control circuitries 510 can be a standalone consoledevice communicatively coupled with other components of system 500. Inthe example shown in FIG. 5, control circuitries 510 include anapplication store 545, a tracking module 550, and an engine 555.

The application store 545 stores one or more applications for executionby the control circuitries 510. An application is a group ofinstructions, that, when executed by a processor, generates content forpresentation to the user. Examples of applications include: gamingapplications, conferencing applications, video playback application, orother suitable applications.

Tracking module 550 tracks movements of near-eye display 100 using, forexample, image date from the imaging device 535. As to be discussed inmore detail below, tracking module 550 may obtain pixel data of an imageframe captured by imaging device 535 and identify one or more objects inthe image frame based on the pixel data, determine the image locationsof the one or more objects across multiple image frames, and determinethe physical locations of the one or more objects based on the imagelocations. In some examples, the physical locations information can beprovided to a SLAM algorithm operated in tracking module 550 to generateposition information of near-eye display 100. In some examples, theimage locations information can be used to generate position informationof the identified object.

Engine 555 executes applications within system 500 and receives positioninformation, acceleration information, velocity information, and/orpredicted future positions of near-eye display 100 from tracking module550. Engine 555 can also receive position information of a physicalobject (other than near-eye display 100) from tracking module 550. Insome embodiments, information received by engine 555 may be used forproducing a signal (e.g., display instructions) to waveguide displayassembly 210 that determines a type of content presented to the user.For example, to provide an interactive experience, engine 555 maydetermine the content to be presented to the user based on a location ofthe user (e.g., provided by tracking module 550), a gaze point of theuser (e.g., based on image data provided by imaging device 535), adistance between an object and user (e.g., based on image data providedby imaging device 535).

FIGS. 6A-6B illustrate examples of operations which can be performed toextend the dynamic range of image sensor 370. Image sensor 370 mayinclude a pixel cell array, which includes pixel cell 600. Pixel cell600 includes a photodiode 602 and a capacitor 604. In the example ofFIG. 6A, image sensor 370 can be used to capture an image of a sceneincluding a light source 606 and an object 608. Due to its locationwithin the pixel cell array, pixel cell 600 may receive high intensitylight from light source 606.

FIG. 6B illustrates a graph 610 which shows the change of quantity ofcharges accumulated at capacitor 604 with respect to time whenphotodiode 602 is exposed to light. As shown in FIG. 6B, at thebeginning (e.g., before time T1) capacitor 604 has the capacity toaccumulate the charges generated by photodiode 602, and the quantity ofcharges increases with time. Before time T1, the quantity of chargesstored at capacitor 604, and the rate at which the quantity of chargesincreases, can be correlated to the intensity of light received byphotodiode 602. On the other hand, after time T1, capacitor 604 becomesfull, and a saturation limit for pixel cell 600 has been reached. As aresult, beyond time T1, capacitor 604 may stop accumulating charges, andthe quantity of charges accumulated in capacitor 604 may remainsubstantially constant. The quantity of charges accumulated in capacitor604 at a time after T1 (e.g., at time T2) may not be correlated to theintensity of light received by photodiode 602. Moreover, beyond time T1,if photodiode 602 remains exposed to the incident light and continuesgenerating charges, those charges may leak into pixel cells neighboringpixel 600 and contaminate the capacitors in those pixel cells, whichcauses blooming as discussed above.

Embodiments of the present disclosure can reduce the likelihood ofsaturating pixel cell 600 (and the resulting blooming effect) bydetecting the onset of saturation at pixel cell 600. The detection ofthe onset of saturation can be based on, for example, determiningwhether the charges stored in capacitor 604 exceed a pre-determinedthreshold. The pre-determined threshold can be set based on a certainpercentage of the saturation capacity of capacitor 604 such that whenthe pre-determined threshold is reached, the pixel cell 600 is notsaturated. Upon determining that the pre-determined threshold isreached, the exposure period can be stopped to prevent photodiode 602from transferring additional charges to capacitor 604. In the example ofFIG. 6B, the exposure period can end before time T1 (e.g., at time T1′).With such arrangements, the likelihood of pixel cell 600 being saturatedcan be reduced. A pixel value can be generated based on, for example,measuring the quantity of charges accumulated in capacitor 604 at timeT1, or measuring the duration of time T1′, which reflects the rate atwhich the charges are accumulated. In both cases, the pixel value canrepresent the intensity of light received by photodiode 602 moreaccurately, and the upper limit of the dynamic range of image sensor 370can be extended.

FIG. 7A illustrates an example of image sensor 370 which includes one ormore pixel cells 700 (e.g., pixel cells 700 a, 700 b, . . . 700 n), anADC 710, a selection module 730, and a controller 740. Pixel cell 700may be part of a pixel array 750, and each pixel cell can generate anoutput (e.g., a voltage output based on the accumulated charges) torepresent an intensity of light upon the pixel cell. Selection module730 can be configured to route the outputs to ADC 710 to generatedigital pixel values based on the outputs, to enable the pixel cells totime-share ADC 710. For example, selection module 730 may be configuredto route the outputs to ADC 710 following a round-robin fashion (e.g.,starting from pixel cell 700 a, followed by pixel cell 700 b, etc.) toenable ADC 710 to process the output of each pixel cell 700 in the pixelcell array. As another example, ADC 710 may include a plurality of ADCs710, each of which corresponds to a pixel cell 700, and selection module730 can be controlled to forward (or not to forward) the output of apixel cell 700 to the corresponding ADC 710. In such an example,selection module 730 and ADC 710 can be part of each of pixel cell 700.

ADC 710 may generate a digital representation of the outputs of thepixel cells. In some examples, as to be discussed in more details below,in a first phase of measurement ADC 710 may generate a digitalrepresentation of a time duration for the quantity of chargesaccumulated in the capacitor to reach a pre-determined threshold. Thefirst phase of measurement can be performed during the exposure period.The pre-determined threshold can be based on a certain percentage of thesaturation capacity of the capacitors of the one or more pre-determinedpixel cells. In the first phase of measurement, ADC 710 may generate anonset of saturation indication based on the quantity of chargesaccumulated in the capacitor reaching the pre-determined threshold forthose pre-determined pixel cells, and transmit the indication tocontroller 740. Moreover, ADC 710 can also perform a second phase and athird phase of measurement, in which ADC 710 can generate a digitalrepresentation of the quantity of charges accumulated in the capacitorof a pixel cell. The second phase of measurement can occur after thefirst measurement phase during the exposure period and the third phasecan occur after the exposure period ends. ADC 710 can performmulti-stage processing of the output from the pixel cell by, forexample, performing the first phase of measurement to detect onset ofsaturation during the exposure period, followed by the second phase andthird phase of measurement. If the onset of saturation is detected, ADC710 may transmit the onset of saturation indication to controller 740.ADC 710 may also transmit the digital representations (generated in thefirst measurement phase or the second measurement phase) to controller740 for further processing.

Controller 740 may include an exposure period setting module 742 and apixel value generation module 744. Exposure period setting module 742may monitor for the onset of saturation indication from ADC 710 for aset of pre-determined pixel cells and, upon receiving the indication,stops the exposure period for all of pixel cells 700 of the pixel array750. The end of the exposure period also stops the first phase ofmeasurement and the second phase of measurement at ADC 710. Moreover, insome examples, controller 740 can skip the third phase of measurement inresponse to receiving the onset of saturation indication. On the otherhand, if no onset of saturation indication is received from ADC 710,exposure period setting module 742 may adopt a pre-set default end timefor the exposure period and ends the exposure period when the defaultend time arrives, and may perform, for example, the first phase ofmeasurement, the second phase of measurement, and the third phase ofmeasurement. The pre-set default end time can be determined based onvarious factors including, for example, the capacity of the capacitor atpixel cell 700, an average ambient light intensity in the environment,the rate of charge generation of the photodiode for the average ambientlight intensity, a rate of motion of the device (e.g., a HMD) thathouses image sensor 370, etc. For example, in a case where the device isundergoing a rapid motion, the default exposure period can be reduced tominimize motion blur in the image. The pre-set default end time can beconfigured such that when the photodiode is exposed to the ambient lightin an exposure period, sufficient spare capacity (e.g., 50%) remains atthe capacitor when the exposure period ends at the pre-set default endtime.

Moreover, pixel value generation module 744 can also control a secondset of pixel cells not selected for onset of saturation detection toperform, for example, the second phase of measurement and the thirdphase of measurement within the same exposure period to obtain thedigital representations of pixel values. Pixel value generation module744 may obtain, from ADC 710, intermediate pixel values provided by thesecond set of pixel cells and perform post-processing based on the onsetof saturation detection from the pre-determined pixel cells. Forexample, in a case where onset of saturation is detected, pixel valuegeneration module 744 can determine a scaling value based on a ratio ofthe default exposure time to the adjusted exposure time (adjusted due toonset of saturation indication), and normalize the intermediate pixelvalues obtained from each pixel cell based on the scaling value. Thenormalization can be used to, for example, deemphasize the image of thecertain regions (e.g., regions with the highest light intensity)relative to the rest of the scene (which is likely to include more imagedetails), to enable an image processor to extract the image details ofthe rest of the scene for other applications (e.g., a location trackingapplication) that rely on those details.

As discussed above, ADC 710 may detect the onset of saturationindication for a set of pre-determined pixel cells, which enablesexposure period setting module 742 to set the end of exposure period forthe rest of the pixel cells of pixel cell array 750. There are differentways by which the set of pixel cells can be selected for onset ofsaturation detection by ADC 710. For example, pixel cell array 750 canbe divided into multiple blocks of pixel cells, and one or more pixelcells can be selected from each block of pixel cells for onset ofsaturation detection. In some examples, selection module 730 can beconfigured to route the output of the same pixel cell(s) to ADC 710 foronset of saturation detection at the beginning of each exposure period.The pixel cell can be selected based on, for example, a location of thepixel cell within the pixel cell array (e.g., at the center of eachblock) which makes it more likely that, for example, the intensity oflight received by that pixel cell represents the block's average lightintensity level. In some examples, selection module 730 can also beconfigured to route the output of different pixel cell(s) to ADC 710 foronset of saturation detection at the beginning of different exposureperiods. For example, due to the movement of the light source and/orimage sensor 370, the pixel cells that exhibit onset of saturation maychange with time, which can be tracked by controller 740. For example,controller 740 may instruct selection module 730 to route the output ofa pixel cell to ADC 710 for onset of saturation detection for a currentexposure period based on that pixel cell exhibiting onset of saturationin the previous exposure period. If that pixel cell no longer exhibitsonset of saturation in the current exposure period, controller 740 mayinstruct selection module 730 not to route the output of that pixel cellto ADC 710 for onset of saturation detection for a subsequent exposureperiod. In some examples, controller 740 may also apply a randomfunction to determine the pixel cell for onset of saturation detection,such that the pixel cell selection can be randomized. All thesetechniques can be adapted to, for example, reduce the likelihood thatcontroller 740 fails to detect onset of saturation in other pixel cellsnot selected for the detection and does not adjust the exposure periodaccordingly.

In some examples, as described above, each pixel cell 700 of imagesensor 370 may also include an ADC 710, and selection module 730 can beomitted (or configured to forward or not to forward pixel cell output toADC 710). In such a case, each pixel cell 700 can stop exposure once thedefault exposure time ends. Moreover, a pre-determined set of pixelcells 700 (e.g., selected based on the techniques described above) canperform a first phase of measurement using its local ADC 710 to generatethe onset of saturation indication, and transmit the indications toexposure period setting module 742. Exposure period setting module 742can stop the exposure period for each pixel cell 700 upon detecting theearliest onset of saturation indication(s) from the pixel cells. On theother hand, other pixel cells 700 that are in the pre-determined set maybe controlled to perform, for example, the second phase and third phaseof measurements to generate digital pixel values representing thatincident light intensity at those pixel cells received during either thedefault exposure period or the adjusted exposure period.

FIG. 7B illustrates an example of a pixel cell 700. As shown in FIG. 7B,pixel cell 700 may include a photodiode 702, a residual charge capacitor703, a shutter switch 704, a transfer gate 706, a reset switch 707, ameasurement capacitor 708, a buffer 709, and a ADC 710.

In some embodiments, photodiode 702 may include a P-N diode or a P-I-Ndiode. Each of shutter switch 704, transfer gate 706, and reset switch707 can include a transistor. The transistor may include, for example, ametal-oxide-semiconductor field-effect transistor (MOSFET), a bipolarjunction transistor (BJT), etc. Shutter switch 704 can act as anelectronic shutter gate (in lieu of, or in combination with, mechanicalshutter 404 of FIG. 4) to control an exposure period of pixel cell 700.During the exposure period, shutter switch 704 can be disabled (turnedoff) by exposure enable signal 711, whereas transfer gate 706 can beenabled (turned on) by measurement control signal 712, which allowscharges generated by photodiode 702 to move to residual charge capacitor703 and/or measurement capacitor 708. At the end of the exposure period,shutter switch 704 can be enabled to steer the charges generated byphotodiode 702 into photodiode current sink 717. Moreover, reset switch707 can also be disabled (turned off) by reset signal 718, which allowsmeasurement capacitor 708 to accumulate the charges and develop avoltage that reflects a quantity of the accumulated charges. The voltagecan be buffered by buffer 709, and the output of buffer 709 (at analogoutput node 714) can be provided to selection module 730, which canroute the output to ADC 710 for processing. After a phase of measurementcompletes, reset switch 707 can be enabled to empty the charges storedat measurement capacitor 708 to charge sink 720, to make measurementcapacitor 708 available for the next measurement. Exposure periodsetting module 742 can control the exposure period and the quantity ofcharges stored at residual charge capacitor 703 and measurementcapacitor 708 by, for example, controlling the timing of shutter switch704, measurement control signal 712, and reset signal 718.

Residual charge capacitor 703 can be a device capacitor of photodiode702 and can store charges generated by photodiode 702. Residual chargecapacitor 703 can include, for example, a junction capacitor at the P-Ndiode junction interface, or other device capacitor(s) connected tophotodiode 702. Due to the proximity of residual charge capacitor 703 tophotodiode 702, charges generated by photodiode 702 may be accumulatedat charge capacitor 703. Measurement capacitor 708 can be a devicecapacitor at a floating terminal of transfer gate 706, a metalcapacitor, a MOS capacitor, or any combination thereof. Measurementcapacitor 708 can be used to store a quantity of charges, which can bemeasured by ADC 710 to provide a digital output representing theincident light intensity. The charges stored at measurement capacitor708 can be either overflow charges (from photodiode 702) that are not tobe accumulated at residual charge capacitor 703, or residual chargesthat are emptied from residual charge capacitor 703.

Reference is now made to FIG. 8, which illustrates the chargeaccumulation operations at residual charge capacitor 703 and measurementcapacitor 708 for different target light intensity ranges. FIG. 8illustrates a total quantity of charges accumulated (or expected toaccumulate) at residual charge capacitor 703 and measurement capacitor708 with respect to time for different light intensity ranges. The totalquantity of charges accumulated can reflect the total charges generatedby photodiode 702 during an exposure period, which in turns reflects theintensity of light incident upon photodiode 702 during the exposureperiod. The quantity can be measured when the exposure period ends. Athreshold 802 and a threshold 804 can be defined for thresholds quantityof charges defining a low light intensity range 806, a medium lightintensity range 808, and a high light intensity range 810 for theintensity of the incident light. For example, if the total accumulatedcharges is below threshold 802 (e.g., Q1), the incident light intensityis within low light intensity range 806. If the total accumulatedcharges is between threshold 804 and threshold 802 (e.g., Q2), theincident light intensity is within medium light intensity range 808. Ifthe total accumulated charges is above threshold 804, the incident lightintensity is within high light intensity range 810.

Thresholds 802 and 804 can be set to control the accumulation of chargesat residual charge capacitor 703 and measurement capacitor 708, toensure that the quantity of accumulated charges at the capacitorscorrelates with the incident light intensity when the incident lightintensity falls within either low light intensity range 806 or mediumlight intensity range 808. For example, thresholds 802 and 804 can beset below the capacities of residual charge capacitor 703 andmeasurement capacitor 708. As discussed above, once residual chargecapacitor 703 and measurement capacitor 708 reaches full capacity, thecapacitors may start leaking charges, and the voltage developed at thecapacitors may not accurately represent or reflect the total number ofcharges generated by photodiode 702 during the exposure period. Bysetting thresholds 802 and 804 to below the capacities of residualcharge capacitor 703 and measurement capacitor 708, measurement errorcaused by charge leakage can be avoided. In some examples, threshold 802can be set at 2000e− (2000 charges), whereas threshold 804 can be set at63000e− (63000 charges).

The accumulation of charges at residual charge capacitor 703 andmeasurement capacitor 708 can be controlled by thresholds 802 and 804.For example, an incident light intensity falling within low lightintensity range 806 can be based on the total charges accumulated atresidual charge capacitor 703. Assuming residual charge capacitor 703 isnot yet full at the end of the exposure period, the total chargesaccumulated at residual charge capacitor 703 can reflect the totalcharges generated by photodiode 702 during the exposure period, and canbe used to determine the incident light intensity. When the totalcharges accumulated at residual charge capacitor 703 exceeds threshold802, the additional charges generated by photodiode 702 can be divertedto measurement capacitor 608 as overflow charges. Assuming measurementcapacitor 708 is not yet full at the end of the exposure period, thetotal overflow charges accumulated at measurement capacitor 708 can alsoreflect the total charges generated by photodiode 702 during theexposure period, and can be used to determine the incident lightintensity (which falls within medium light intensity range 808).

On the other hand, in a case where the incident light intensity iswithin high light intensity range 810, the total overflow chargesaccumulated at measurement capacitor 708 may exceed threshold 804 beforethe exposure period ends. As additional charges are accumulated,measurement capacitor 708 may reach full capacity before the end of theexposure period, and charge leakage may occur. To avoid measurementerror caused due to measurement capacitor 708 reaching full capacity, atime-to-saturation measurement can be performed to measure the timeduration it takes for the total overflow charges accumulated atmeasurement capacitor 708 to reach threshold 804. A rate of chargeaccumulation at measurement capacitor 708 can be determined based on aratio between threshold 804 and the time-to-saturation, and ahypothetical quantity of charge (Q3) that could have been accumulated atmeasurement capacitor 708 at the end of the exposure period (if thecapacitor had limitless capacity) can be determined by extrapolationaccording to the rate of charge accumulation. The hypothetical quantityof charge (Q3) can provide a reasonably accurate representation of theincident light intensity within high light intensity range 810.Moreover, with embodiments of the present disclosure, the exposureperiod can be adjusted to shift leftwards to time T′ when the quantityof accumulated charges reaches threshold 804, to reduce the likelihoodof saturating the pixel cell.

Referring back to FIG. 7B, transfer gate 706 can be controlled bymeasurement control signal 712 to control the charge accumulations atresidual charge capacitor 703 and measurement capacitor 708 fordifferent light intensity ranges as described above. For example, forlow light intensity range 806, transfer gate 706 can be controlled tooperate in a partially turned-on state. During the exposure period, thegate voltage of transfer gate 706 can set based on a voltage developedat residual charge capacitor 703 when the total accumulated charges atresidual charge capacitor 703 reaches threshold 802. With sucharrangements, the charges generated by photodiode 702 will be stored inresidual charge capacitor 703 first until the quantity of accumulatedcharges reaches threshold 802. Right before the exposure period ends,transfer gate 706 can be controlled to operate in a fully turned-onstate to move the charges stored in residual charge capacitor 703 tomeasurement capacitor 708. At the end of the charge transfer, transfergate 706 can be controlled to operate in a fully turned-off state topreserve the charges stored in measurement capacitor 708. At that point,the charges stored in measurement capacitor 708 may represent thecharges stored in residual charge capacitor 703, and can be used todetermine the incident light intensity. On the other hand, for mediumlight intensity range 808 and high light intensity range 810, theoverflow charges accumulated at measurement capacitor 708 can also bemeasured right before the exposure period ends, when the transfer gate706 is still in a partially turned-on state and the charges stored inresidual charge capacitor 703 are not yet transferred to measurementcapacitor 708.

The charges accumulated at measurement capacitor 708 can be sensed bybuffer 709 to generate a replica of the analog voltage (but with largerdriving strength) at analog output node 714. The analog voltage atanalog output node 714 can be converted into a set of digital data(e.g., comprising logical ones and zeros) by ADC 710. The analog voltagedeveloped at measurement capacitor 708 can be sampled and digital outputcan be generated before the end of the exposure period (e.g., for mediumlight intensity range 808 and high light intensity range 810), or afterthe exposure period (for low light intensity range 806). The digitaldata can be transmitted by a set of pixel output buses 716 to, forexample, control circuitries 510 of FIG. 5, to represent the lightintensity during the exposure period.

In some examples, the capacitance of measurement capacitor 708 can beconfigurable to improve the accuracy of light intensity determinationfor low light intensity ranges. For example, the capacitance ofmeasurement capacitor 708 can be reduced when measurement capacitor 708is used to measure the residual charges stored at residual chargecapacitor 703. The reduction in the capacitance of measurement capacitor708 can increase the charge-to-voltage conversion ratio at measurementcapacitor 708, such that a higher voltage can be developed for a certainquantity of stored charges. The higher charge-to-voltage conversionratio can reduce the effect of measurement errors (e.g., quantizationerror, comparator offset, noise associated with the buffer circuit,etc.) introduced by ADC 710 on the accuracy of low light intensitydetermination. The measurement error can set a limit on a minimumvoltage difference that can be detected and/or differentiated by ADC710. By increasing the charge-to-voltage conversion ratio, the quantityof charges corresponding to the minimum voltage difference can bereduced, which in turn reduces the lower limit of a measurable lightintensity by pixel cell 700 and extends the dynamic range. On the otherhand, for medium light intensity, the capacitance of measurementcapacitor 708 can be increased to ensure that the measurement capacitor708 has sufficient capacity to store a quantity of charges up to, forexample, the quantity defined by threshold 804.

FIG. 9 illustrates an example of the internal components of ADC 710. Asshown in FIG. 9, ADC 710 includes a threshold generator 902, acomparator 904, and a digital output generator 906. Digital outputgenerator 906 may further include a counter 908 and a memory 910.Counter 908 can generate a set of count values based on a free-runningclock signal 912, whereas memory 910 can store at least some of thecount values (e.g., the latest count value) generated by counter 908. Insome embodiments, memory 910 may be part of counter 908. Memory 910 canbe, for example, a latch circuit to store the counter value based onlocal pixel value as described below. Threshold generator 902 includes adigital-to-analog converter (DAC) 913 which can accept a set of digitalvalues and output a reference voltage (VREF) 915 representing the set ofdigital values. As to be discussed in more detail below, thresholdgenerator 902 may accept static digital values to generate a fixedthreshold, or accept output 914 of counter 908 to generate a rampingthreshold.

Although FIG. 9 illustrates that DAC 913 (and threshold generator 902)is part of ADC 710, it is understood that DAC 913 (and thresholdgenerator 802) can be coupled with multiple digital output generators906 from different pixel cells. Moreover, digital output generator 906(and ADC 710) can also be shared among a plurality of multiple pixelcells to generate the digital values.

Comparator 904 can compare the analog voltage developed at analog outputnode 714 against the threshold provided by threshold generator 902, andgenerate a decision 916 based on the comparison result. For example,comparator 904 can generate a logical one for decision 916 if the analogvoltage at analog output node 714 equals to or exceeds the thresholdgenerated by threshold generator 902. Comparator 904 can also generate alogical zero for decision 916 if the analog voltage falls below thethreshold. Decision 916 can control the counting operations of counter908 and/or the count values stored in memory 910, to perform theaforementioned time-of-saturation measurement of a ramping analogvoltage at analog output node 714 as well as quantization processing ofthe analog voltage at analog output node 714 for incident lightintensity determination.

FIG. 10A illustrates an example of time-to-saturation measurement by ADC710. To perform the time-to-saturation measurement, threshold generator802 can control DAC 813 to generate a fixed VREF 915. Fixed VREF 915 canbe set at a voltage corresponding to a charge quantity threshold betweenthe medium light intensity range and the high light intensity range(e.g., threshold 804 of FIG. 8). Counter 908 can start counting rightafter the exposure period starts (e.g., right after shutter switch 704is disabled). As the analog voltage at analog output node 714 ramps down(or up depending on the implementation), clock signal 912 keeps togglingto update the count value at counter 908. The analog voltage may reachthe fixed threshold at a certain time point, which causes decision 916by comparator 904 to flip. The flipping of decision 916 may stop thecounting of counter 908, and the count value at counter 908 mayrepresent the time-to-saturation, with a smaller count value indicatinga higher rate of charge accumulation and a higher light intensity.Decision 916 of FIG. 10A can also be used to provide an indication ofonset of saturation, in a case where the voltage at analog output node714 is compared against a pre-determined threshold representing afraction of the saturation limit.

FIG. 10B illustrates an example of quantizing an analog voltage by ADC710. After measurement starts, DAC 913 may be programmed by counteroutput 914 to generate a ramping VREF 915, which can either ramp up (inthe example of FIG. 10B) or ramp down depending on implementation. Inthe example of FIG. 10B, the quantization process can be performed withuniform quantization steps, with VREF 915 increasing (or decreasing) bythe same amount for each clock cycle of clock signal 912. The amount ofincrease (or decrease) of VREF 915 corresponds to a quantization step.When VREF 915 reaches within one quantization step of the analog voltageat analog output node 714, decision 916 by comparator 904 flips. Theflipping of decision 916 may stop the counting of counter 908, and thecount value can correspond to a total number of quantization stepsaccumulated to match, within one quantization step, the analog voltage.The count value can become a digital representation of the quantity ofcharges stored at measurement capacitor 708, as well as the digitalrepresentation of the incident light intensity. As discussed above, thequantization of the analog voltage can occur during the exposure period(e.g., for medium light intensity range 808) and after the exposureperiod (e.g., for low light intensity range 806).

As discussed above, ADC 710 can introduce quantization errors when thereis a mismatch between a quantity of charges represented by the quantitylevel output by ADC 710 (e.g., represented by the total number ofquantization steps) and the actual input quantity of charges that ismapped to the quantity level by ADC 710. One way to reduce quantizationerror can be by employing a non-uniform quantization scheme, in whichthe quantization steps are not uniform across the input range. FIG. 10Cillustrates an example of a mapping between the ADC codes (the output ofthe quantization process) and the input charge quantity level for anon-uniform quantization process and a uniform quantization process. Thedotted line illustrates the mapping for the non-uniform quantizationprocess, whereas the solid line illustrates the mapping for the uniformquantization process. For the uniform quantization process, thequantization step size (denoted by Δ₁) is identical for the entire rangeof input charge quantity. In contrast, for the non-uniform quantizationprocess, the quantization step sizes are different depending on theinput charge quantity. For example, the quantization step size for a lowinput charge quantity (denoted by Δ_(S)) is smaller than thequantization step size for a large input charge quantity (denoted byΔ_(L)). Moreover, for the same low input charge quantity, thequantization step size for the non-uniform quantization process (Δ_(S))can be made smaller than the quantization step size for the uniformquantization process (Δ₁).

One advantage of employing a non-uniform quantization scheme is that thequantization steps for quantizing low input charge quantities can bereduced, which in turn reduces the quantization errors for quantizingthe low input charge quantities, and the minimum input charge quantitiesthat can be differentiated by ADC 710 can be reduced. Therefore, thereduced quantization errors can push down the lower limit of themeasurable light intensity of the image sensor, and the dynamic rangecan be increased. Moreover, although the quantization errors areincreased for the high input charge quantities, the quantization errorsmay remain small compared with high input charge quantities. Therefore,the overall quantization errors introduced to the measurement of thecharges can be reduced. On the other hand, the total number ofquantization steps covering the entire range of input charge quantitiesmay remain the same (or even reduced), and the aforementioned potentialproblems associated with increasing the number of quantization steps(e.g., increase in area, higher bandwidth requirement, reduction inprocessing speed, etc.) can be avoided.

FIG. 10D illustrates an example of quantizing an analog voltage by ADC710 using a non-uniform quantization process. Compared with FIG. 10B(which employs a uniform quantization process), VREF 915 increases in anon-linear fashion with each clock cycle, with a shallower slopeinitially and a steeper slope at a later time. The differences in theslopes are attributed to the uneven quantization step sizes. For lowercounter count values (which correspond to a lower input quantity range),the quantization steps are made smaller, hence VREF 915 increases at aslower rate. For higher counter count values (which correspond to ahigher input quantity range), the quantization steps are made larger,hence VREF 915 increases at a higher rate. The uneven quantization stepsin VREF 915 can be introduced using different schemes. For example, asdiscussed above, DAC 913 is configured to output voltages for differentcounter count values (from counter 908). DAC 913 can be configured suchthat the difference in the output voltage between two neighboringcounter count values (which defines the quantization step size) isdifferent for different counter count values. As another example,counter 908 can also be configured to generate jumps in the countercount values, instead of increasing or decreasing by the same countstep, to generate the uneven quantization steps. In some examples, thenon-uniform quantization process of FIG. 10D can be employed for lightintensity determination for low light intensity range 806 and mediumlight intensity range 808.

Reference is now made to FIG. 11, which illustrates an example of pixelcell 1100, which can be an embodiment of pixel cell 700 of FIG. 7. Inthe example of FIG. 11, PD can correspond to photodiode 702, transistorMO can correspond to shutter switch 704, transistor M1 can correspond totransfer gate 706, whereas transistor M2 can correspond to reset switch707. Moreover, PDCAP can correspond to residual charge capacitor 703,whereas a combination of COF and CEXT capacitors can correspond tomeasurement capacitor 708. The capacitance of measurement capacitor 708is configurable by the signal LG. When LG is enabled, measurementcapacitor 708 provides combined capacities of COF and CEXT capacitors.When LG is disabled, CEXT capacitor can be disconnected from theparallel combination, and measurement capacitor 708 comprises only COFcapacitor (plus other parasitic capacitances). As discussed above, thecapacitance of measurement capacitor 708 can be reduced to increase thecharge-to-voltage conversion ratio for the low light intensitydetermination, and can be increased to provide the requisite capacityfor the medium light intensity determination.

Pixel cell 1100 further includes an example of buffer 709 coupled withan example of ADC 710 via a selection module (not shown in FIG. 11). Forexample, transistors M3 and M4 form a source follower which can bebuffer 709 of FIG. 7B to buffer an analog voltage developed at the OFnode, which represents a quantity of charges stored at the COF capacitor(or at the COF and CEXT capacitors). Further, the CC cap, comparator1110, transistor M5, NOR gate 1112, together with memory 910, can bepart of ADC 710 to generate a digital output representing the analogvoltage at the OF node. As described above, the quantization can bebased on a comparison result (VOUT), generated by comparator 1110,between the analog voltage developed at the OF node and VREF. Here, theCC cap is configured to generate a VIN voltage (at one input ofcomparator 1110) which tracks the output of buffer 709, and provides theVIN voltage to comparator 1110 to compare against VREF. VREF can be astatic voltage for time-of-saturation measurement (for high lightintensity range) or a ramping voltage for quantization of an analogvoltage (for low and medium light intensity ranges). The ADC code inputcan be generated by a free-running counter (e.g., counter 908), and thecomparison result generated by comparator 1110 can determine the ADCcode input to be stored in memory 910 and to be output as the digitalrepresentation of the incident light intensity. In some examples, thegeneration of VREF for low and medium light intensity determination canbe based on a non-uniform quantization scheme as discussed in FIG. 10Cand FIG. 10D.

Pixel cell 1100 includes techniques that can further improve theaccuracy of the incident light intensity determination, in addition tothe techniques disclosed above. For example, the combination of the CCcap and transistor M5 can be used to compensate for measurement errors(e.g., comparator offset and various noise) introduced by comparator1110, as well as other error signals that are introduced to comparator1110, such that the accuracy of comparator 1110 can be improved. Thenoise signals may include, for example, reset noise charges introducedby reset switch 707, a noise signal at the output of buffer 709 due tosource follower threshold mismatches, etc. A quantity of chargesreflecting the comparator offset as well as the error signals can bestored at the CC cap during a reset phase, when both transistors M2 andM5 are enabled. A voltage difference can also be developed across the CCcap during the reset phase due to the stored charges. During ameasurement phase, the voltage difference across the CC cap remains, andthe CC cap can track the output voltage of buffer 709 by subtractingaway (or adding) the voltage difference to generate VIN. As a result,the VIN voltage can be compensated for the measurement errors and theerror signals, which improves the accuracy of the comparison between VINand VREF and the ensuing quantization.

In some examples, pixel cell 1100 can be operated in either athree-phase measurement process or a two-phase measurement process. Thethree-phase measurement process can be used for identifying textures orother image features of a physical object, such as hand 642. Each of thethree phases can correspond to one of the three light intensity rangesof FIG. 8 (e.g., low light intensity range 806, medium light intensityrange 808, and high light intensity range 810). On the other hand, thetwo-phase measurement process can be used for identifying an image of anobject or an event generated from light of low light intensity range 806and light of high light intensity range 810, but not from light ofmedium light intensity range 808. The two-phase measurement process canbe used for, for example, detection of pixel data corresponding to glintpatch 632 (which are likely in high light intensity range 810) and pixeldata corresponding to pupil 634 (which are likely in low light intensityrange 806), detection of dot patterns of infra-red lights (which arelikely in high light intensity range 810), etc. By skipping one phase ofmeasurement, the total duration of the measurement process can bereduced, which allows both power consumption and output latency to bereduced. Moreover, by using a set of bits to represent the low lightintensity range 806 instead of a combined (and larger) range includinglow light intensity range 806 and medium light intensity range 808, thequantization error can be reduced, and the accuracy of detecting objectsof low light intensity (e.g., pupil 634) can be improved as a result.

In each phase of the three-phase or two-phase measurement processes,pixel cell 1100 can be operated in a measurement phase targeted for thecorresponding light intensity range, and determine whether the incidentlight intensity falls within the corresponding light intensity rangebased on the output of comparator 1110. If the incident light intensityfalls within the corresponding light intensity range, pixel cell 1100can latch the ADC code input (from counter 908) into memory 910, and puta lock (using a combination of the FLAG_1 and FLAG_2 signals) on memory910 to prevent subsequent measurement phases from overwriting memory910. At the end of the two-phase or three-phase measurement processes,the ADC code input stored in memory 910 can then be provided as thedigital output representing the incident light intensity.

Reference is now made to FIGS. 12A-12D, which illustrate the change ofthe control signals of pixel cell 1100 for a three-phase measurementprocess with respect to time. Referring to FIG. 12A, the time periodbetween T0 and T1 corresponds to a first reset phase, whereas the timeperiod between T1 and T4 corresponds to the exposure period. Within theexposure period, the time period between T1 and T2 corresponds to afirst phase of measurement for a high light intensity range (e.g., highlight intensity range 810), the time period between T2 and T3corresponds to a second phase of measurement for a medium lightintensity range (e.g., medium light intensity range 808), whereas thetime period between T3 and T4 corresponds to a second reset phase.Moreover, the time period between T4 and T5 corresponds to a third phaseof measurement for a low light intensity range (e.g., low lightintensity range 806). Pixel cell 1100 can provide the digital outputrepresenting the incident light intensity at time T5, and then start thenext three-phase measurement process.

As shown in FIG. 12A, during the time period between T0 and T1 both RST1and RST2 signals, as well as the LG signal, the TX signal, and theshutter signal, are asserted. As a result, the charges stored in thePDCAP capacitor, the CEXT capacitor, and the COF capacitor are removed.Moreover, no charges are added to the capacitors because the chargesgenerated by photodiode PD are diverted away by transistor MO. Further,comparator 1110 is also in a reset phase, and the CC capacitor can storecharges reflecting the reset noise introduced by M2, the comparatoroffset, the threshold mismatch of buffer 709, etc. Towards the end ofthe period, the TX gate is biased at a threshold level to trap apre-determined number of charges (e.g., threshold 802) at PDCAP. Thethreshold level can be set based on a voltage corresponding to thepre-determined number of charges.

During the time period between T1 and T2, the shutter signal isde-asserted and the LG signal remains asserted, which allow the chargesgenerated by the PD photodiode to flow into the PD capacitor and, if thevoltage developed at the PD capacitor exceeds the threshold level set byTX, to flow into the COF capacitor and the CEXT capacitor. FIG. 12Billustrates the measurement operations performed by ADC 710 during thattime period. As shown in FIG. 12B, ADC 710 can perform thetime-to-saturation measurement, and the buffered and error-compensatedversion of analog voltage at the OF node (VIN) can be compared against athreshold voltage representing a quantity of charges of threshold 804while counter 908 is free-running. If the total charges stored at theCOF capacitor and the CEXT capacitor exceeds threshold 804 (based on theOF node voltage), comparator 1110 will trip, and the count valuegenerated by counter 908 at the time of tripping can be stored intomemory 910. The tripping of comparator 1110 also causes a registerstoring FLAG_1 to store a value of 1. The FLAG_1 signal can also be sentto controller 740 to represent an onset of saturation indication. Thenon-zero FLAG_1 value can cause the output of NOR gate 1112 to remainlow regardless of other inputs to the NOR gate, and can lock the memoryand prevent subsequent measurement phases from overwriting counter 908.On the other hand, if comparator 1110 never trips during the time periodbetween T1 and T2, which indicates that the incident light intensity isbelow the high light intensity range, FLAG_1 stays zero. FLAG_2, whichcan be updated by subsequent measurement phases, stays at zeroregardless of whether comparator 1110 trips.

During the time period between T2 and T3, which corresponds to thesecond phase of measurement, the analog voltage at the OF node can bequantized by ADC 710. FIG. 12C illustrates the measurement operationsperformed by ADC 710 during that time period. As shown in FIG. 12C, aramping VREF can be supplied to comparator 1110 to be compared againstthe buffered and error-compensated version of analog voltage at the OFnode (VIN). Although FIG. 12C shows a ramping VREF corresponding to auniform quantization process, it is understood that the ramping VREF canalso include non-uniform slopes corresponding to a non-uniformquantization process as described with respect to FIG. 10B. The secondphase of measurement ends at T3, by which time the ADC input codesrepresenting the entire medium incident light range have been cycledthrough. If the ramping VREF matches VIN (within one quantization step),comparator 1110 will trip, and the count value generated by counter 908at the time of tripping can be stored into memory 910, if the memory isnot locked by the first phase of measurement (as indicated by the zerovalue of FLAG_1). If the memory is locked, the count value will not bestored into memory 910. On the other hand, if the memory is not locked,the count value generated by counter 908 at the time of tripping can bestored into memory 910, and the memory can be locked by writing a valueof 1 to a register storing FLAG_2.

At the beginning of the time period between T3 and T4, both RST1 andRST2 signals can be asserted again for a second reset phase. The purposeof the second reset phase is to reset the CEXT and COF capacitors, andto prepare the COF capacitor for storing charges transferred from thePDCAP capacitor in the third phase of measurement (for low lightintensity range). The LG signal can also be de-asserted to disconnectthe CEXT capacitor from the COF capacitor and to reduce the capacitanceof measurement capacitor 708. The reduction of the capacitance is toincrease the charge-to-voltage conversion ratio to improve the low lightintensity determination, as discussed above. Comparator 1110 is also putinto the reset state where the CC cap can be used to store the noisecharges generated by the resetting of the CEXT and COF capacitors.Towards time T4, after the resetting completes, the RST1 and RST2signals are de-asserted, whereas the bias TX can increase to fully turnon transistor M1. The charges stored in the PD cap can then move intothe COF capacitor via M1. As discussed above, exposure period settingmodule 742 may set the exposure period by, for example, setting thetiming of T4 based on receiving an asserted FLAG_1 signal whichindicates the pixel cell is in the onset of saturation.

During the time period between T4 and T5, the third phase of measurementis performed for the low light intensity range. During that period, theshutter signal is asserted to end the exposure period, whereas the TXsignal is de-asserted to disconnect the PDCA capacitor and the PDphotodiode from the COF capacitor, to ensure that the COF capacitor,during the time of measurement, only stores the charges stored in thePDCAP capacitor during the exposure period. FIG. 12D illustrates themeasurement operations performed by ADC 710 during that time period. Asshown in FIG. 12D, a ramping VREF can be supplied to comparator 1110 tobe compared against the buffered and error-compensated version of analogvoltage at the OF node (VIN). Although FIG. 12D shows a ramping VREFcorresponding to a uniform quantization process, it is understood thatthe ramping VREF can also include non-uniform slopes corresponding to anon-uniform quantization process as described with respect to FIG. 10D.The third phase of measurement ends at T5, by which time the ADC inputcodes representing the entire low incident light range have been cycledthrough. As shown in FIG. 12D, if the ramping VREF matches VIN (withinone quantization step), comparator 1110 will trip, and the count valuegenerated by counter 908 at the time of tripping can be stored intomemory 910, if the memory is not locked by the first and second phasesof measurement (as indicated by the zero values of FLAG_1 and FLAG_2).If the memory is locked, the count value will not be stored into memory910. The count value stored in memory 910 can then be provided as adigital output representing the incident light intensity.

Reference is now made to FIG. 13, which illustrate the change of thecontrol signals of pixel cell 1100 for a two-phase measurement processwith respect to time. Referring to FIG. 13, the time period between T0and T1 corresponds to a first reset phase, whereas the time periodbetween T1 and T2 corresponds to the exposure period. Within theexposure period a first phase of measurement for a high light intensityrange (e.g., high light intensity range 810) can be performed. Moreover,the time period between T2 and T3 corresponds to OF reset and PD chargetransfer, whereas the time period between T3 and T4 corresponds to asecond phase of measurement for a low light intensity range (e.g., lowlight intensity range 810), with the measurement phase for medium lightintensity range skipped. The operations of pixel cell 1100 in each phaseof measurement are similar to as discussed with respect to FIG. 12B andFIG. 12D, the detail of which are not repeated here.

FIG. 14 illustrates an embodiment of a flowchart of a process 1400 fordetermining incident light intensity at a pixel cell (e.g., pixel cell700, pixel cell 1100, etc.). Process 1400 can be performed by acontroller together with various components of pixel cell 700 and pixelcell 1100. Process 1400 begins in step 1402, where the pixel cell isoperated in an exposure mode where the photodiode can transfer chargesto the residual charge capacitor and/or the measurement capacitor. Instep 1404, the pixel cell can be operated to compare a voltage developedat the measurement capacitor with a fixed threshold voltage to generatea first decision and a first count at the counter. Step 1404 can be thefirst measurement phase targeted for a high light intensity range, andthe first count can represent a time-of-saturation measurement. If thefirst decision is positive (in step 1406), the pixel cell can proceed tostep 1408 and store the first count in a memory and then lock the memoryby asserting a first flag (e.g., FLAG_1), and then proceed to step 1410to perform the second measurement phase. If the first decision is notpositive, the pixel cell can also proceed directly to step 1410 toperform the second measurement phase.

In step 1410, the pixel cell is operated to compare the voltagedeveloped at the measurement capacitor with a first ramping voltage togenerate a second decision and a second count at the counter. Step 1410can be the second measurement phase for a medium light intensity range.The pixel cell can then determine whether the second decision ispositive, and whether the memory is not locked (e.g., based on firstflag, FLAG_1 remains de-asserted), in step 1412. If the second decisionis positive and the memory is not locked, the pixel cell can proceed tostep 1414 and store the second count in the memory and lock the memoryby asserting a second flag (e.g., FLAG_2), and then proceed to step 1416to perform the third measurement phase. If the first decision is notpositive, the pixel cell can also proceed directly to step 1416 toperform the third measurement phase.

In step 1416, as part of the third measurement phase, the pixel cell canreset the measurement capacitor to empty the stored charges. The pixelcell can also reduce the capacitance of the measurement capacitor toincrease the charge-to-voltage conversion ratio, in step 1418. In step1420, the pixel cell can transfer residual charges stored in theresidual charge capacitor of the photodiode to the measurementcapacitor. The pixel cell then proceeds to step 1422 to compare thevoltage developed at the measurement capacitor with a second rampingvoltage to generate a third decision and a third count at the counter.The pixel cell then proceeds to determine whether the third decision ispositive and whether the memory is not locked (e.g., based on whetherany one of first flag FLAG_1 and second flag FLAG_2 is asserted), instep 1424. If the third decision is positive and the memory is notlocked, the pixel cell stores the third count in the memory in step1426, and then proceeds to step 1428 to output the count value stored inthe memory. On the other hand, if the third decision is not positive, orthe memory has been locked, the pixel cell will proceed directly to step1428 to output the count value (which can be one of the first count orthe second count) stored in the memory. In some examples, step 1428 canbe performed at a later time, and step 1428 need not be performedimmediately following step 1424 or 1426.

FIG. 15 illustrates an embodiment of a flowchart of a process 1500 foroperating an array of pixel cells. The array of pixel cells may include,for example, an array of pixel cell 700, an array of pixel cells 1100,etc. Process 1500 can be performed by a controller (e.g., controller740) that interfaces with the array of pixel cells and with an ADC(e.g., ADC 710). Each pixel cell may perform, in conjunction with theADC, process 1400 of FIG. 14. Process 1500 begins in step 1502, wherethe controller starts an exposure period of a pre-set duration to enablean array of pixel cells to accumulate charges generated based onincident light. The pre-set duration may be configured based on, forexample, the capacity of the capacitor at pixel cell 700, an averageambient light intensity in the environment, the rate of chargegeneration of the photodiode for the average ambient light intensity,etc. The exposure period can be started by, for example, opening amechanical shutter to expose the pixel cells to incident light,disabling a charge steering switch (e.g., shutter switch 704), and/orenabling a charge transfer switch (e.g., transfer gate 706), etc., toenable the capacitors at the pixel cells to accumulate charges.

In step 1504, the controller determines whether a quantity of chargesaccumulated by at least one pixel cell of the array of pixel cellsexceeds a pre-determined threshold. The pixel cell array may be dividedinto multiple blocks of pixel cells, and the at least one pixel cell mayinclude one pixel cell selected from each block of pixel cells. The sameor different pixel cells can be selected for different exposure periods.The pre-determined threshold can be based on a fraction of a saturationlimit of the capacitor of the at least one pixel cell. The determinationof whether the quantity of charges accumulated by the at least one pixelcell of the array of pixel cells exceeds the pre-determined thresholdcan be based on, for example, an onset of saturation indication (e.g.,FLAG_1 of FIG. 11) generated by a comparator.

If the quantity of charges exceeds the threshold (in step 1506), thecontroller may proceed to step 1508 to stop the accumulation of chargesat each pixel cell of the array of pixel cells. On the other hand, ifthe quantity of charges does not exceed the threshold (in step 1506),the controller may proceed to step 1510 to determine whether the pre-setduration has expired. If the pre-set duration has not expired, thecontroller may proceed back to step 1504 to continue monitoring foronset of saturation at the at least one pixel cell. If the pre-setduration has expired, the controller may also proceed to step 1508 tostop the accumulation of charges at each pixel cell of the array ofpixel cells. The stopping of the accumulation of charges can beperformed by, for example, closing the mechanical shutter to shield thepixel cells from incident light, enabling the charge steering switch(e.g., shutter switch 704), and/or disabling a charge transfer switch(e.g., transfer gate 706), etc., to prevent the capacitors at the pixelcells from accumulating charges.

In step 1512, the controller may receive (e.g., from ADC 710), anintermediate pixel value for each pixel cell based on the accumulatedcharges at the each pixel cell of the array of pixel cells. Theintermediate pixel values may be generated based on thetime-to-saturation output in the first phase of measurement, or based onthe quantity of charges stored at the capacitor in the second or thirdphases of measurements.

In step 1514, the controller may determine a scaling value for scalingthe intermediate pixel values. The scaling value can be determined basedon, for example, a ratio between the duration of the adjusted exposureperiod and the duration of the default exposure period.

In step 1516, the controller may scale each of the intermediate pixelvalues using the scaling value to generate a set of digital pixelvalues, and provide the digital pixel values to an image processor forgeneration of an image frame in step 1518.

The foregoing description of the embodiments of the disclosure has beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the disclosure to the precise forms disclosed.Persons skilled in the relevant art can appreciate that manymodifications and variations are possible in light of the abovedisclosure.

Some portions of this description describe the embodiments of thedisclosure in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are commonly used by those skilled in the dataprocessing arts to convey the substance of their work effectively toothers skilled in the art. These operations, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as modules, withoutloss of generality. The described operations and their associatedmodules may be embodied in software, firmware, and/or hardware.

Steps, operations, or processes described may be performed orimplemented with one or more hardware or software modules, alone or incombination with other devices. In some embodiments, a software moduleis implemented with a computer program product comprising acomputer-readable medium containing computer program code, which can beexecuted by a computer processor for performing any or all of the steps,operations, or processes described.

Embodiments of the disclosure may also relate to an apparatus forperforming the operations described. The apparatus may be speciallyconstructed for the required purposes, and/or it may comprise ageneral-purpose computing device selectively activated or reconfiguredby a computer program stored in the computer. Such a computer programmay be stored in a non-transitory, tangible computer readable storagemedium, or any type of media suitable for storing electronicinstructions, which may be coupled to a computer system bus.Furthermore, any computing systems referred to in the specification mayinclude a single processor or may be architectures employing multipleprocessor designs for increased computing capability.

Embodiments of the disclosure may also relate to a product that isproduced by a computing process described herein. Such a product maycomprise information resulting from a computing process, where theinformation is stored on a non-transitory, tangible computer readablestorage medium and may include any embodiment of a computer programproduct or other data combination described herein.

The language used in the specification has been principally selected forreadability and instructional purposes, and it may not have beenselected to delineate or circumscribe the inventive subject matter. Itis therefore intended that the scope of the disclosure be limited not bythis detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsis intended to be illustrative, but not limiting, of the scope of thedisclosure, which is set forth in the following claims.

What is claimed is:
 1. An apparatus comprising: an array of pixel cells,each pixel cell including: a photodiode configured to generate chargesupon receiving incident light; and a capacitor configured to accumulatethe charges generated by the photodiode; and a controller configured to:start an exposure period to enable the capacitors of the array of pixelcells to accumulate the charges; determine whether a quantity of chargesaccumulated by at least one pixel cell of the array of pixel cellsexceeds a pre-determined threshold; and based on a determination thatthe quantity of charges accumulated by the at least one pixel cellexceeds a pre-determined threshold: end the exposure period to cause thecapacitors of the array of pixel cells to stop accumulating the charges,generate an output pixel value for each pixel cell based on the chargesaccumulated at the capacitor of the each pixel cell within the exposureperiod; and provide the output pixel values for generation of an imageframe.
 2. The apparatus of claim 1, wherein the controller is furtherconfigured to: determine an intermediate pixel value for each pixel cellbased on the charges accumulated at the capacitor of the each pixel cellwithin the exposure period; determine a scale value based on a durationof the ended exposure period; and scale each of the intermediate pixelvalues using the scale value to generate the output pixel values.
 3. Theapparatus of claim 1, wherein the pre-determined threshold is set basedon an intensity range of incident light that saturates the at least onepixel cell.
 4. The apparatus of claim 1, wherein the pre-determinedthreshold is set based on a capacity of the capacitor of the at leastone pixel cell for accumulating the charges.
 5. The apparatus of claim1, further comprising: one or more analog-to-digital converters (ADC)configured to generate a digital pixel value based on at least one of: ameasurement of time for the capacitor of a pixel cell to accumulate aquantity of charges equal to the pre-determined threshold, or ameasurement of the quantity of charges accumulated at the capacitor whenthe exposure period ends; and a selection module configured to coupleeach pixel cell of the array of pixel cells sequentially to the one ormore ADCs to generate the digital pixel value for the each pixel cellbased on the charges accumulated at the photodiode of the each pixelcell.
 6. The apparatus of claim 5, wherein the controller is configuredto, in each exposure period of a plurality of exposure periods: select apixel cell from the array of the pixel cells as the at least one pixelcell; control the selection module to couple the selected pixel cell tothe one or more ADCs to determine whether a quantity of chargesaccumulated at the selected pixel cell exceeds the pre-determinedthreshold; and responsive to determining that the quantity of chargesaccumulated at the selected pixel cell exceeds the pre-determinedthreshold, end the each exposure period.
 7. The apparatus of claim 6,wherein the controller is configured to select the same pixel cell inthe each exposure period of the plurality of exposure periods.
 8. Theapparatus of claim 6, wherein the controller is configured to selectdifferent pixel cells in a first exposure period and a second exposureperiod of the plurality of exposure periods.
 9. The apparatus of claim6, wherein the controller is configured to select the pixel cell in acurrent exposure period based on the digital pixel value of the pixelcell exceeding the pre-determined threshold in a prior exposure period.10. The apparatus of claim 6, wherein the controller is configured toselect the pixel cell based on a random function.
 11. The apparatus ofclaim 1, wherein each pixel cell includes an analog-to-digital converter(ADC) configured to generate a digital pixel value for the each pixelcell based on at least one of: a measurement of time for the capacitorof the each pixel cell to accumulate a quantity of charges equal to thepre-determined threshold, or a measurement of the quantity of chargesaccumulated at the capacitor when the exposure period ends; and whereinthe controller is configured to: monitor for an indication that aquantity of charges accumulated at at least one of the pixel cellsexceeds the pre-determined threshold; and end the exposure period foreach pixel cell based on receiving the indication.
 12. The apparatus ofclaim 1, wherein the exposure period has a default end time; wherein thecontroller is configured to end the exposure period before the defaultend time based on the determination that the quantity of chargesaccumulated by the at least one pixel cell exceeds the pre-determinedthreshold; and wherein the default end time is preset based on anambient light intensity.
 13. The apparatus of claim 1, wherein the arrayof pixel cells is a first array of pixel cells; wherein the apparatusfurther comprises a second array of pixel cells; and wherein thecontroller is configured to: start the exposure period at a first timefor the first array and for the second array; end the exposure period ata second time for the first array; and end the exposure period at athird time different from the second time for the second array.
 14. Amethod comprising: starting an exposure period to enable a capacitor ofeach pixel cell of an array of pixel cells to accumulate chargesgenerated by a photodiode included in the each pixel cell; determiningwhether a quantity of charges accumulated by at least one pixel cell ofthe array of pixel cells exceeds a pre-determined threshold; and basedon determining that the quantity of charges accumulated by the at leastone pixel cell exceeds a pre-determined threshold: ending the exposureperiod to cause the capacitors of the array of pixel cells to stopaccumulating the charges; generating an output pixel value for eachpixel cell based on the charges accumulated at the capacitor of the eachpixel cell within the exposure period; and providing the output pixelvalues for generation of an image frame.
 15. The method of claim 14,further comprising: determining an intermediate pixel value for eachpixel cell based on the charges accumulated at the capacitor of the eachpixel cell within the exposure period; determining a scale value basedon a duration of the ended exposure period; and scaling each of theintermediate pixel values using the scale value to generate the outputpixel values.
 16. The method of claim 14, wherein the pre-determinedthreshold is set based on an intensity range of incident light thatsaturates the at least one pixel cell.
 17. The method of claim 14,wherein the pre-determined threshold is set based on a capacity of thecapacitor of the at least one pixel cell for accumulating the charges.18. The method of claim 14, further comprising: generating, using anADC, a digital pixel value for each pixel cell based on at least one of:a measurement of time for the capacitor of the each pixel cell toaccumulate a quantity of charges equal to the pre-determined threshold,or a measurement of the quantity of charges accumulated at the capacitorof the each pixel cell when the exposure period ends.
 19. The method ofclaim 18, further comprising: in each exposure period of a plurality ofexposure periods: selecting a pixel cell from the array of the pixelcells as the at least one pixel cell; controlling the ADC to determinewhether a quantity of charges accumulated at the selected pixel cellexceeds the pre-determined threshold; and responsive to determining thatthe quantity of charges accumulated at the selected pixel cell exceedsthe pre-determined threshold, ending the each exposure period.
 20. Themethod of claim 19, wherein the same pixel cell is selected in the eachexposure period of the plurality of exposure periods.
 21. The method ofclaim 19, wherein different pixel cells in a first exposure period areselected in a first exposure period and in a second exposure period ofthe plurality of exposure periods.
 22. The method of claim 19, whereinthe pixel cell is selected in a current exposure period based on thedigital pixel value of the pixel cell exceeding the pre-determinedthreshold in a prior exposure period.
 23. The method of claim 19,wherein the pixel cell is selected based on a random function.